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(57) Abstract: An apparatus that automatically 
captures, stores and analyzes images of crystallization 
experiments contained in a number of crystallization 
plates. The apparatus includes a plate nest capable 
of accommodating protein crystallization plates of a 
plurality of different types, image acquisition optics, 
including an objective lens and an image capturing 
device, for focusing an image of a crystallization well, 
a light source including a bright field illumination 
device and a dark field illumination device, a nest 
positioning controller for moving the position of the 
plate nest with respect to the image acquisition optics 
to align various selected wells with said objective lens 
for imaging of the content of the wells. A database 
stores experiment information associated with each of 
the crystallization plates, the experiment information 
including identification of specific crystal forming 
parameter values, each of the crystallization plates is 
identified in said database by a unique identification 
code. A crystallization imaging controller controls 
crystallization imaging by retrieving the experiment 
information for each crystallization plate inserted into 
the apparatus, and controlling the nest positioning 
controller and the image acquisition optics in 
accordance with the retrieved experiment information. 
The apparatus captures multiple images of each crystal 
site using different light source and polarization 
conditions, and processes the multiple images to form 
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Automated Protein Crystallization Imaging 



Cross-Reference to Related Application 

5 The present application claims priority under 35 U,S.C, § 1 19(e) of provisional 

application Serial No. 60/371,1 15, filed April 10, 2002, and incorporated herein by 
reference in its entirety. 

■ 

Background of the Invention 

The present invention relates generally to the field of protein crystalhzation axialysis 
10 and in particular to a process and apparatus for capturing, indexing, analyzing and 
storing images of protein crystallization samples. 

Protein crystallization is a technique used to analyze the molecular structure of 
proteins. In tins technique, varying controlled ratios and combinations of certain 
15 chemical reactants or catalysts are added to a protein sample or "drop" to induce the 
protein to crystallize so that its structure may be analyzed for use in biochemical 
research applications. 

As shown in Fig. 1, one type of crystallization plate includes a number of ceUs, each 
20 cell having a number of satellite wells (three in the example). The cells are identified 
by row and colunm number, such as Al, B 1 0, etc., and each satellite well is identified 
by its position number in ttie cell (such as 1, 2 or 3). A quantity of a protem is placed 
into each of the satellite wells, and a different experimental amount of a 
ciystallization catalyst is added to each satellite well in an attempt to induce 
25 crystallization of the protein. The mixture is given a sufficient time to react, and then 
the satellite wells must be viewed under a microscope to determine if sufficient 
crystallization has occurred. The particular amounts and/or ratios of crystallization 
catalyst material and protein placed in each sateUite well must be recorded in order to 
determine which ratios and combinations are successful in producing crystallizations. 
30 Typically, one protein will be used for each plate, and different catalysts and ratios of 
catalysts will be added to the protein drops in each satellite well. 

Achieving successfiil protein crystallization is very time consmning and tedious, as 
only a very small nmnber of crystallizations of a protein occiu: for many tens or 

35 hundreds of thousands of experimental reactant ratios and combinations. 

Additionally, the contents of a well may be more complex than a simple crystal 
structure. Specifically, the protein sample (drop) may be turbid as opposed to clear, 
there may be micro-crystal fields in the sample, protein may be aggregated, and 
crystals may be embedded in a precipitate field. Since each of these possibihties 

40 requires specialized Ughting conditions in order to be detected, it is necessary to take 
many different pictures of the same well under different lighting and focus conditions, 
thus adding even more complexity to the analysis process. 

Accordingly, there is a need for improvement in obtaining, analyzing, and storing 
45 protein crystallization experiment images whereby all potential information in an 
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experiment well can be captured quickly and efficiently, and stored together with all 
experimental parameters and conditions associated with the well. 



Summary of the invention 

5 

According to a preferred embodiment of the invention, an apparatus is provided that 
automatically captures, stores and analyzes images of crystallization experiments 
contained m a plurality of crystallization plates, the apparatus including a plate nest 
capable of accomanodating protein crystallization plates of a plurality of different 

10 types, the plates each including a plurality of crystalUzation wells each including a 

particular crystallization experiment, image acquisition optics^, including an objective 
lens and an image capturing device, for focusing an image of a crystallization well 
positioned under the objective lens and electronically capturing the focused image, a 
light source including a bright field illumination device and a dark field illumination 

15 device, each of the bright field and dark field illumination devices being 

independently energized to introduce light into a crystal image well to enable capture 
of separate bright field and dark field images of the crystal image well by the image 
acquisition optics, a nest positioning controller for moving the position of the plate 
nest with respect to the image acquisition optics to ahgn various selected wells with 

20 the objective lens for imaging of the content of the wells, a database for storing 

experiment information associated with each of the plurality of crystalhzation plates, 
the experiment information including identificatibn of specific crystal fonning 
parameter values, and wherein each of the plurality of crystallization plates is 
identified in the database by a unique identification code, and a crystallization 

25 ima^g controller for controlhng crystallization imaging by retrieving the 

information for each crystallization plate inserted into the apparatus, and controlling 
the nest positioning controller and the image acquisition optics in accordance with the 
retrieved experiment information. 

30 According to another aspect of the invention, the imager apparatus obtains and stores 
multiple images of each desired crystallization well under a number of different 
lighting and/or focusing conditions, and processes the multiple images to form a fused 
image containing all available information pertaining to a crystal image well. 

35 Brief Description of the Drawings 

Fig. 1 is a diagram of a protein crystallization plate for use in an. imager according to 
the present invention; 

Fig. 2 is a diagram of a combination imaging Ught source according to one 
40 embodiment of the invention using LEDs; 

Fig. 3 is a diagram of a combination imaging light source according to another 
embodiment of the invention using an external halogen light source; 

45 Fig. 4 is a diagram showing the use of polarization filters in combination with a tight 
source according to another aspect of the invention; 
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Fig. 5 a shows a series of narrow focus images of a protein crystal takeji at different 
focus depths, and Fig. 5b shows a composite extended focus image obtained by 
combining the narrow focus images of Fig. 5a, according to another aspect of the 
invention; 

5 

Fig. 6 is a flow diagram of one embodiment of a process for obtaining the merged 
extended focus image of Fig. 5b according to the present invention; 

Fig. 7 is a flow diagram of one embodiment of a process for detection of protein drops 
10 on a "hanging drop" plate according to Ihe present invention; 

Fig. 8 is a perspective view of an imager housing showing a plate inserted into a plate 
nest holder according to one embodiment of the invention; 

15 Fig. 9 is a perspective view of the plate nest holder showing cut outs to accormnodate 
a robot plate gripper; 



20 



35 



Fig. 10 is a perspective view of the imager light source configurations according to 
one embodiment of the invention; 

Figs. 1 1 and 12 are respective views of a light polarizer filter in a disengaged position 
and an engaged position; 



Fig. 13 is a perspective view of the imaging optics components according to one 
25 embodiment of the invention; 

Figs. 14 and 15 are perspective views showing accommodation of different imaging 
plate types by the plate nesting holder according to one embodiment of the invention; 

30 Fig. 16 is a perspective view showing an XY servo stage connected to the plate nest to 
enable precise weU positioning control with respect to the imaging optics; 



Fig. 17 is a perspective view showing plate accommodation details of a plate nesting 
holder according to one embodiment of the invention; 

Fig. 1 8 is a perspective view showing plate centering cams integrated into the plate 
nesting holder according to one embodiment of the invention; 



Figs. 19 and 20 are views showing the operation of the plate centering cams to 
40 automatically center a misaligned portrait oriented plate in the nesting holder; 

Figs. 21 and 22 are views showing the operation of the plate centering cams to 
automatically center a misaligned landscape oriented plate in the nesting holder; 

45 Figs. 23 and 24 are views showing the operation of a Z-axis pressure sensor to detect 
pressure exerted on an imaging plate as it is engaged with the imagmg optics, to 
prevent damage to the imaging plate; and 

Fig, 25 is a screen shot of a viewing and scoring computer application according to 
50 one embodiment of the invention. 
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Detailed Description of the Preferred Embodiments 

The present invention will now be described in conjunction with the accompanying 
5 . drawings. However, it will be understood that the following detailed description is 
for purposes of providing an explanation and disclosure of the concepts of the 
invention only, and is not intended to define the scope of the invention, which is 
defined solely by the claims. 

1 0 The imager has a plate nest or holder, as shown in Fig. 8, which extends to a 

load/unload position via a servo controller as shown, to permit an imaging plate to be 
loaded into the nest for imaging, and to be removed from the nest after image 
processing is completed. The loading and unloading of the plates may be performed 
manually, or via robotics jSrom a plate storage area. Each plate is labeled with a bar 

1 5 code label containing a imique identification code, which allows automatic 

identification of each plate through scanning of the bar code label by a bar code 
scanner. After a plate is loaded, the nest is retracted into the imager for image 
processing. 

20 The imager has the ability to be run in one of three modes: automatic, semi-automatic, 
or manual. In automatic mode, each plate is unaged predicated on the image mask 
parameters selected by the user on a GUI (graphical user interface, see Fig. 25) during 
a plate setup phase. An image mask is the marking of the wells from which images 
are desired. 

25 

In the example plate shown in Fig. 1 , cells Al, B 10, D4 and F9 are marked by the 
user to be imaged. Each cell of the plate contains three satellite wells. The satellite or 
sitting drop well is selected based on its position (1, 2, and/or 3 in this example). 
Depending on which satellite well setting is desired, the imager will automatically 
30 acquire images for the selected wells per the preselected satellite well selection. Once 
all images have been captured the nest is extended to the load/unload position located 
outside of the main housing, as shown in Figure 8. 

In a semi-automatic mode, each plate is associated with an image mask as in the 
35 automatic mode, but the operator is allowed to make adjustments to the image 

acquisition process such as contrast, magnification, focus, Ught configuration, etc., for 
each image position. After the plate has been positioned to acquire an image, a real- 
time image is displayed on the operator's interface as shown in Fig, 25 and the 
operator may adjust the settings as desired. Once the operator is satisfied with the 
40 image settings, the image is captured and the operator then can either acquire 

additional images of the same well, or may choose to move to the next image location 
that has been preselected in the mask settings. Once all image sites per tihie mask have 
been processed, the plate nest is extended to the load / unload position where the plate 
is removed. 

45 

In the manual mode, the plate is loaded manually, and there is no image mask 
associated with it. The operator manually selects the desired satellite wells or 
positions to image and is able to set the same imager configuration settings as those 
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mentioned in the semi-automatic mode. The operator controls plate loading and 
unloading before and after image processing is performed. 

The imager accommodates aU presently known protein crystalUzation plate formats. 
The imager plate nest supports all SBS (Society for Biomolecular Screening) format 
plates with heights ranging from the thinnest glass shde capable of supporting drops 
(approximately 0.1 inch) to 1.6 inches as shown in Fig. 15. The nest also supports 
large-well LinbroAHDX plate sizes as shown in Fig. 14. 

The imager stage, shown in Fig. 16, is a three axis servo-controlled device which 
allows motion in the plate horizontal plane (X and Y axes) and in the camera vertical 
plane (Z axis). Control of each axis is independent of the other axes, allowing an 
infinite range of positions, which is necessary to accommodate all of the various plate 
formats that are commercially available. The stage positioning resolution allows the 
smallest well format to be accurately positioned for imaging. It has been found that 
by linking the camera auto focus routine to the Z axis via a mathematical algorithm 
focus compensation during zoom operations is accelerated. In addition to the plate 
nest stage, the variable transmitting polarizer and the lens focus mechanism also are 
servo controlled. 

The imager nest is shown in detail in Fig. 17. The nest is designed to handle SBS and 
large-well Linbro formatted plates as explained above. As shown, the nest is 
designed to have a reUef to allow the maximum amount of light to pass through the 
wells located on the edge of the plate. The plates are held in tlie nest by a number of 
nest fingers as shown, such that the plates "ride" above the nest platform. It has been 
found that the reUeved nest mitigates shadowing effects on the wells located near the 
edge of the plate, thereby dramatically improving image quahty. 

As shown in Fig. 15, the nest includes an active plate orienting system. This system 
uses a multiple cam mechanism which forces the plate into the correct orientation 
regardless of plate placement in the nest As shown in Figures 19 and 20, the 
camming mechanism automatically centers plates inserted in a portrait orientation, 
and as shown in Figures 21 and 22, the camming mechanism also automatically 
centers plates inserted into the nest in a landscape orientation. The cam locating pins 
are opened as the plate nest is extended into the load position and are then closed 
when the nest is retracted into the main housing of the imager. Correct plate 
orientation is necessary to provide the required accuracy to locate and image the plate 
at high magnifications. The active cam mechanism provides an automated 
methodology to correctly orient all plate types. 

In a manual loading mode, the imager nest is folly extended with the cam centering 
devices open. A plate is placed in the nest either in portrait and landscape fashion 
depending on the plate type by hand. The operator instructs the imager via the GUI to 
begin imaging. Chice the image operation is completed for that plate the nest is 
extended so that the plate is presented to the operator as shown in Figure 8 with the 
plate centering cams open. The operator removes the plate by hand completing the 
manual loading and unloading process. 

In the automatic and semi-automatic modes, the imager may be loaded via robotics. 
The nest is commanded to extend via a series of conununications from a high level 
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robot controller. Once extended the imager reports back to the controller that it is 
ready to load and the robot then places the plate in the imager in either portrait or 
landscape orientation depending on the plate type. As shown in Fig. 9, the nest has 
cut outs specifically located to allow the robot grippers to move down into the nest 

5 while still gripping the plate. The plate is placed in the nest and the giippers are 

opened. The robot retracts and sends a signal to the imager that it is clear of the nest 
and that the plate is located in the nest. The imager retracts the nest, which causes the 
plate centering cams to engage the plate and center the plate. Once the plate imaging 
operation has been completed, the nest is extended and the imager signals the robot 

1 0 controller that the plate is ready for retrieval. The plate cams are opened as the plate 
is extended. The robot retrieves the plate in the same manner as it loaded it by 
dropping into the nest along the reheved cut-out portions. 

The imager has the ability to acquire multiple high-resolution iraages of each well 

15 position using multiple various well lighting techniques. The multiple image data is 
then fused to obtain a more complete and accurate understanding of the well contents. 
Thus, the imager is a true multi-mode device that can analyze a well and determine 
more information about the well than is possible from a single image event. 
Specifically the imager can determine crystal presence, clear versus turbid drop 

20 contents^ micro-crystal field, crystals embedded in a precipitate field, and protein 

aggregation. Using bright field backlighting crystal detection is enhanced, while dark 
field lightuig enhances precipitate detection, and polarized light can be used to 
enhance micro-crystal detection. Additionally, due to the properties of the various 
fluids used in protein crystallization, the drops in sitting well experiments can form a 

25 convex or concave shape, which diverts light either away or toward the unaging 
microscope objective lens, causing imdesirable shadowing effects. The dark field 
light mitigates these shadowing effects by changing the incidence angle of the light on 
the well. The dark field light will illuminate areas of the well that were not 
illuminated by the bright field light due to this shadowing effect. Thus the 

30 combination of these two lights will mitigate the shadowing effects. The imager also 
provides for the means to oversize the bacWigJit and thus floods the imaging area with 
Ught of various vectors, which has been found to also mitigate this shadowing effect. 
It has been found that no one single image can fully represent the artifacts of a protein 
crystallization drop in all cases and therefore a multunode image process is required 

35 to produce acciurate results in optical protein crystallization analysis. 

As shown in Fig. 10, the imager uses three distinct lighting techniques: bright field 
illumination, dark field illumination, and variable angle polarized light illumination. 
Each light souix;e has a distinct purpose as discussed above. Fig. 10 shows the 

40 orientation of each lighting component in the imager. Each lighting technique is 
controlled separately and can be energized independent of the other sources by a 
processing controller. This elumnates multiple light effects when collecting multi- 
mode data. The imager housing enclosure shown in Fig. 8 provides a light tight 
environment as well, which eliminates or substantially reduces undesired effects of 

45 ambient Hght entering the imaging plate wells. 

The bright field illuminating technique is used as a backlight and illuminates the drop 
using a transmitted Ught approach. When employing this technique the polarizing 
filter transmitter is disengaged as shown in Figure 1 1 , by rotating the polarizing filter 
50 so that a notch in the filter aligns with the bright field light source. The light is a 
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special high-intensity low heat generation light. The heat load generated by 
transmitted and reflected light sources is mitigated through various techniques, 
including the use of air cooling provided by a fan in the enclosure housing, as well as 
through the use of heat sink technology incorporated into the light housing, and most 
importantly by controlling the activation of light sources independently, thereby 
allowing the light sources to be energized only when acquiring an image. The 
independent control of the light sources also provides for the ability to vary intensity 
as well as to energize and de-energize each light source without impacting other light 
sources in the imager. 

The bright field light or back light is built using LUMI-LED technology which offers 
longer lasting life, has the ability to be energized and de-energized many times 
without shortening the expected hfe and bums very cool as compared to a standard 
halogen source. The LUMI-LED teclmology offers a great increase in intensity over 
standard LED technology making the LUMI-LED a significant improvement over 
standard lighting techniques. It has been found that LUMI-LED technology provides 
a more stable back light source for protein crystallization imaging with improved 
control and longer lasting life. 

The dark field technique utilizes a standard dark field light source, which has been 
found to enhance certain aspects of a protein drop image by changing the angle of 
incidence of the light with respect to the protein crystal drop. The dark field source 
can be either an LED source with independent control or a standard halogen source 
using a dark field adapter. The imager provides a means to use either type of lighting. 

As shown in Fig. 2, the bright field and dark field Ught sources can be implemented 
using LED or LUMI-LED technology, and controlled by a dual-light controller via 
LED control cables.The LED dark field source provides a stable long life illumination 
source for protein crystallization with the added benefit of being able to independently 
control the light (e.g. vary intensity, energize and de-energize the source without 
effecting other light sources). This allows the dark-field light to be de-energized 
during back light operations and energized for dark field operations exclusively (i.e., 
the bright field light is not energized). This arrangement allows for a clear and 
distinct light combmation to be used in the multi-mode approach prescribed by the 
imager, which, is advantageous in producing higher quality protein crystal drop 
images. 

Alternatively, as shown in Fig. 3 an external halogen light source can be used in 
concert with a fiber optic dark field adapter. The benefit of this combination is the 
generation of very uniform and intense dark field illumination without the addition of 
the halogen light heat source internal to the imaging enclosure tod the resulting 
negative impact to the protein crystallization experiment, which requires a very 
uniform enviroimxent during the storage and time-lapsed imaging process. The 
external halogen light source may remain on during all operational modes, including 
bright field and polarized light illumination. 

In protein crystallography polarized light in concert with transnaitted white Ught and a 
color camera can provide details that are not always- discemable to the human eye 
even with the aide of magnification. Because a property of most protein crystals is bi- 
refidngence, polarized light can be used to find crystals that are either to small to be 
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distinguished by normal processing methods or crystals that may have become 
occluded in a precipitate field. As shown in Fig. 4 and also in Figs. 11 and 12, a 
rotating transmitter polarizing filter is provided between the light source and the 
imaging plate, and an analyzer polarizing filter is placed between the imaging plate 
and the imaging optics. 

The physics of light travel is such that if a polarizer transmitter and analyzer are set to 
extinction angles that no fight will pass; therefore, it has been found that if a protein 
crystal is not perfectly aUgned with either the transmitter or analyzer axes then the 
fight rays wiU be redirected by the presence of a protein crystal so that light will pass 
to the imaging optics with the analyzer and transmitter set at extinction, provided that 
the weU is not bi-refiingent. In this condition only crystals will be seen; however 
because many protem crystal plates are composed of materials that during formation 
also exhibit bi-refiingent properties a clean hnage is not returned during imaging with 
the analyzer and transmitter at extinction, fiistead the plate itself will redirect the Ught 
as it passes through the plate by changing the angle of the fight to a myriad of axes 
other than that of the transmitter. The result is a distinct diffiise fight signature at the 
analyzer. In order to mitigate these plate effects, it has been found that by varying the 
transmitter polarizing angles by rotation of the filter and acquiring multiple images at 
these various angles and then by transforming the negative of the image and 
perfomiing image subtraction on the resultmg set of unages, the polarized composite 
result displays a higher level of crystal confidence and shape than is possible with a 
fixed polarizer transmitter / analyzer angle. The orientation of the filters with respect 
to the object under inspection is critical and must be set so that the transmitter is on 
the fight source side of the plate and (he analyzer is on flie objective or camera side of 
the plate as depicted in Fig. 4 and also shown in Fig. 13. 

In order to completely filter out Ught, a technique of employmg two polarizers set at 
90 degrees with respect to each other is used. This first polarizer filter passes light 
30 in only one direction or axis with respect to the angle of the polarizer, this filter is 
referred to as the transmitter. Therefore with the imager setup shown in Fig. 12, 
when the transmitter filter is engaged (ie, inserted between the light source and the 
imaging plate), fight is passed m one axis. This fight passes through the crystal plate 
and then travels to the microscope objective. Prior to entering the objective the fight 
3 5 passes through the analyzer polarizmg filter, which in the case of the imager doubles 
as a color filter. In order to accommodate the requkement of variable position based 
on user desires the transmitter is attached to a servomotor via a non-slip belt and 
pulley assembly. This arrangement provides complete positional control of the 
transmitter angle with respect to the protem drop to be nnaged. The filter is built by 
40 laser etching it mto an opticaUy transparent disk and then notchmg out a portion of the 
disk so that it can be disengaged when bright field and dark field imagmg are desired, 
as shown in Figs. 1 1 and 12. The position-contioUed transmitter polarizing filter 
enhances image processing and data coUection in protein crystaUization drop analysis. 

45 As described above, a complete polarizmg solution requkes a transmitter polarizer 
and an analyzer polarizer. The imager accompfishes this by utifizing the polarizing 
filter aspect of a color filter, which is located between the microscope objective and 
the camera as shown m Figure 13. It has been found that an LCD color filter has the 
correct optical characteristics to act as an analyzing polarizer filter for protein 

50 crystallization imagmg. In addition to fimctioning as the analyzer filter of the 
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polarizing system, the color filter provides the capabiUty to acquire true color images 
without requiring a multiple CCD camera. The color filter allows three images to be 
taken with a single CCD array digital camera as shown in Fig. 13, and then combined 
to form a composite true color image. While other single CCD color cameras without 
the use of a color filter employ a mosaic approach known as the Bayer Pattem to 
simulate a color image, the Bayer Pattem does not accurately define a tme color 
image because of the redundant green pixel pattem used. 

Turbidity Analysis 

The imager supports the ability to conduct light scattering analysis to calculate 
turbidity of the protein crystallization drop. The is achieved by measiuing the amount 
of light collected over an extended time fimne and then filtering the data set to 
determine Ught scattering patterns. It has been found that early stages, prior to crystal 
formation of protein aggregation can be detected using this methodology. The 
analysis can be conducted using a variety of standard algorithms such as Discrete 
Fourier Transform, Fast Fourier Transform, Neural Net Filter, and Statistical Filters 
(such as LQE, LQR, LMS, etc.). 

The turbidity detection methodology uses a Hght scattering or collection technique to 
determine a turbidity measure of the well. The imager can utihze different 
approaches to conducting this methodology. Two such methods are described below 
for completeness; however, by changing the Ught source and collection device the 
imager can support other approaches to turbidity analysis. 

According to one embodiment, the imager can utilize a laser, which is tuned to the 
appropriate wavelength and mimics the approximate size of the object under analysis 
(in the case of the imager, a protein crystal drop). The laser beam is transnodtted 
through the drop and a collector on the opposing side of the drop measures the 
transmitted light level. The more turbid the drop, the greater the resulting Ught 
attenuation. This measurement is taken over a period of time and is coupled with the 
image data in order to show in clear to semi-clear drops early stages of protein 
aggregation as previously explained. The laser Ught passed through the crystal plate 
well is measured for intensity over a period of time so that any deviation can be 
mapped over flie sampling period and correlated with corresponding image data in 
order to detect early stages of protein aggregation. 

The imager also can employ the abiUty of a digital camera to measure Ught intensity 
while employing bright field, dark field, or laser Ughting techniques. The premise is 
identical to that stated above using a laser emitter collector approach. Thus, 
according to an alternate embodiment, the Ught intensity focused through a well can 
be measured over a period of time using the CCD array of the digital camera to 
capture the light intensity variations. 

As shown in Fig. 13, the imager uses a true microscope objective to enhance image 
depth of field (DOF) and provide the abiUty to focus through a very large range of 
field of views (FOV). The result of this type of optics provides a large range of 
magnifications that can be used by the imager. It has been found that this approach 
allows for more dynamic range of FOV, DOF and magnification than a standard set of 
optics with a variable magnification lens. 
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The imager employs a digital camera to capture a high resolution image and to allow 
digital image processing to occur. The camera employs a high speed data transfer 
ability to move the data from the camera to the associated user computer (not shown). 
The camera has the capability to merge three single color images of the same object 
into one composite true color image. 

The camera produces unages which can vary in resolution dependent on user desures. 
The higher the resolution of the rniage, the mcreased likelihood of small object 
detection; however, the image data size also increases with resolution. It has been 
found that lossy compressed data unages can be manually scored, but a lossless 
compression is required for automatic scoring accuracy. Therefore, accordmg to a 
preferred embodiment, the imager saves a lossy version of the image for rapid manual 
scoring and a lossless version for automated scoring in various image formats which 
are user selectable. 

The imager has the ability to acquire full color or monochrome images at a high 
resolution. The maager allows the operator to select the mode and resolution of the 
images to be taken in semi-auto mode or ahead of tune for tlie entire image set run in 
auto mode. 

In Fig. 1 3 a Z Axis pressure sensor is shown mounted at the end of the microscope 
objective. This sensor is used to detect the presence of the top of a plate contained in 
the nest without applying i force to the plate. The sensor has a compUant ring that 
rests below the horizontal plane of the microscope objective (lens) as shown in Fig. 
23. As the sensor comes in contact with a plate as shown in Fig. 21, the sensor ring is 
pushed upward. As the sensor is displaced, a focused beam of light to a optical 
detector that produces a signal when the light beam is obstracted. This detector sends 
a digital signal to the imager servo control system indicating an anomaly in the Z 
position of the lens. The z-axis servo travel in the positive or down direction is then 
blocked to avoid a crash scenario, thus protecting the plate and the objective. 

The imager also utilizes a vibration damped baseplate that is constructed usmg a 
honeycomb composite. This plate design is lighter than Granite damping plates and 
affords a ready methodology to mount the imager components to the baseplate 
without requiring special purpose tools and fasteners. The vibration damped 
baseplate provides real time platform stability isolating stage movement during image 
acquisition of protein crystallization plates and therefore mitigates settling periods 
required for image capture after plate movement is complete. 

Software Components 

The imager utilizes a sophisticated set of software components to control, acquire, and 
analyze the protein crystalUzation imaging process. The imager control systeni reads 
specific instruction sets for each plate via a database (not shown). The instructions 
are associated with the plate barcode so that each plate has a macro associated with 
the experiment analysis set to be run for that particular plate. This methodology 
allows for unique e^qierimental analysis steps to be followed on each plate 
independent of other plates in the system at that or any previous time. The following 
is an explanation of the various software modules that are utilized in the imager. 



10 



• 



wo 03/087326 PCT/US03/10954 



The imager acquires images digitally and transfers tbe images jfrom the camera via a 
high speed data transfer medium such as IEEE 1394, Fast SCSI, or a camera 
proprietary portal to an image processing PC. The images are associated with the 
plate barcode and well identification code (e.g. Al) in the database by having the 
control system identify the plate barcode either robotically or manually depending on 
the operation and then associating the well ID fi^om the servo positioning software 
with the acquired image or multiple images of each well. All of this information is 
saved together with the actual image location in the database. The imager uses an 
image mask to acquire the designated images for a particular plate as described 
previously, winch image mask also is stored in the database. The camera is triggered 
to acquire the image fi^om the imager control system software once the plate is 
correctly positioned imder the lens, which information is fed back to the controller 
fi-om the servo positioning algorithms (which are well known and therefore will not 
be further described herein). The images are either concatenated in the digital camera 
before being transmitted to the control system computer or are sent to tlie controller 
and are processed and merged into a high resolution composite image at that point. 
The image processing and system control algorithms used by the imager are described 
below. 

The imager has the abiUty to automatically merge multiple images of narrow depth of 
field as shown in Fig, 5A, into a single extended focus image as shown in Fig. 5B. In 
this example fom: images of a protein drop with crystal grov^ are merged into one 
extended focused image. The process to create the merged image requires four steps, 
which are described in Fig. 6. As eadh image is acquired, a median filter is applied to 
a copy of the image to remove any noise in die image. The median-filtered image is 
then processed with a two dimensional edge detector (Sobel detector) to create an 
edge image that is placed in a local maximum map. After all desired images have 
been acquired and processed, each edge image is searched on a pixel-by-pixel basis 
within a variable neighborhood to determine which image has the greatest edge value. 
The image with the greatest edge value then contributes its original image pixel to the 
final merged image. The final merged image is then constmcted pixel-by-pixel usmg 
the process described above. 

The imager also has the ability to automatically focus the image currently in the view. 
This is a two step process. When the user actuates the auto focus capability the 
imager begins a search routine using standard deviation of the image as a measure. A 
second order curve fit is used to predict the position corresponding to the maximum 
standard deviation of the image. A second search is performed using a smaller range 
and step size and using a Tenengrad measure in place of the standard deviation. The 
maximum Tenengrad value is used as the focused position. In addition the auto focus 
routine can use a region of interest to determine the optunum focused position 
allowing the user to focus on a specific region of the image. 

The imager further has the ability to automatically find the optimum exposure for the 
hnage. The camera has an integral capability to rapidly sample a range of exposure 
levels and select the value for the unage witii the lowest saturation level. By default 
auto exposure works with the entire image but a region of interest can be specified. 
The imager also has the abihty to automatically set Ihe white balance for the image. 
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The camera has an integral capability to sample a user selected pixel in the image and 
adjust the individual color balance to force that pixel to be white. 

To determine whether polarization is to be used the imager control system software 
5 r °ai SSSe using the plate barcode and detemiines the necessary expemnent 
se?£gS.an£rze the plate. Once the plate is loaded in the imager flie polar^er is 
SnedT&e desi^ angle and an image is acquired. Multiple unages at various 
Szer angles can be taken if the user has requested this in the experiment setup 
eITpoI^^ image is saved and associated with a single unage event for the plate 
10 and the well in the database. These images can be used to provide a more 

!^moreWiveviewofthewellcontentsbasedonthevanons As 
. SL'^tdTvioisly most protein plates have a bi-reftingent property and therefore 
multiple polarization angles can provide more data to enhance acciiracy. 

1 ^ Th^ irri^per can save images m any format where a defined codec is avaUable under 
S Xdows^eSiriystem. Such known codecs include JPG. BMP. PNG -d 
TO W o&^image encoding formats that provide similar quahty and capabihty 
I^o cm be ^ed according to the mvention. The imager may use uncompressed BMP 
foSitraSt wWch to conduct automated scoring (image processmg) techmques. 

20 S^uSectstheformattobeusedforimagestobe^^&.^^^ 

These formate may employ an image compression algonthm that will greatly 
accelerate the aimotation and manual viewing process. 

For turbidity analysis, the imager associates the results with the plates and well 
25 !osi^lftStabkse. The time lapsed results for a single weU are jhen Passed 

CuSlfil^ which estimates and predicts light scattering or absorption by the well 
S ite conteSs The algorithm takes the data pointe for a well for each image event 
^d SLs^SL vak^s inl a statistical filter to calculate the deviation of the measur^ 
3,t over tiie course of tiie recorded image events, hi addition the values are 

30 iSfstiX^SfdlfutureeventeasweUaspredictedusinga^ 
These values are used to predict early stages of protem aggregation. 

TTie data related to a particular well on an imaging plate is stor^ in the database, and 
S W data then is collated or fused by an engine which perfoms a series of 
•^S SiiSandrule-basedprocessestomoreaccuratelydiscernwellcontentsand 

ScSewellconditi^ons. Hxese results become the final score associated to the 
we?2 Se^atZse while all of the original well data is stiU mamtamed By fiismg 
r dSa sete ^ociated with a weU and applying basic statistical metiiods. adaptive 
filt^g tSSiies and custom rule-based processes, a more accurate assessment of 

40 the well contents can be made. 



45 



The imager can use image processing algorithms to automafacally find crystals and 

in weU images. These algorithms coupled witii the various «nages taken m 
L various different modes can be used to more accurately f J^^J^^^J . „ 

^.ntents for protein crystalUzation. The image processing algontims utihze -blob 

tSques, morphological techniques, edge, and centiroid detecti^ techm^ 
to fiiU? qualiftr the well contents from a digital image processmg approach. By using 
stand^d advanced image processing softw^ ^^onthms clear dropd^^ 
precipitate recognition, and crystal recogmtion can be achieved. Furthermore early 
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stages of protein aggregation can be modeled and identified using advanced statistical 
methods such as digital filter and adaptive filter methods. Such statistical and filtering 
algorithms are well known in the art and therefore will not be further described 
herein. 

The imager supports manual annotation and viewing of the images talcen by the 
system in one of two modes: offline post-image acquisition and onhne real-time 
annotation during the acquisition cycle. Figure 25 depicts one of the interfaces 
supported by the manual image annotation software. In this figure the image viewing 
area allows a detailed look at the complete image as well as the ability to zoom to a 
region of interest either digitally (post image acquisition) or zoom and acquire a new 
image (real-time during ttie image acquisition process). The viewing window also 
supports measurement tools that are associated with rulers that encompass the 
viewing window or by defining a region of interest on the screen via a pointing device 
such as a mouse, using standard practices adhering to the Windows Operating System 
Style Guide. The imaging area can also be changed to view multiple images at one 
time in a thumbnail format. These images can be either time lapse views of a single 
well on a particular plate or images &om a common plate that were taken during a 
single image event (i.e., at the same relative time). The location of each image in its 
perspective well is shown in the plate map. Fiuthermore the plate map also shows all 
of the previous image events for a particular plate across the top of the plate map in 
the form of a tabs which show the date and tune of the image event. The previous 
image events can be viewed in detail mode by selecting the desired tab on this 
interface. The images can also be annotated via this interface in either offline or 
online modes. The annotation field shown in Figure 25 is user configurable and is 
established from the database to which the users have access in order to modify via 
another apphcation. The annotation names and modifiers can therefore be customized 
by each user. The user can select the desired annotation and modifier to associate 
with a particular image, and also has the option to describe the image in a narrative 
format The resulting annotations are associated with the image in the database via 
the plate barcode and well location fields. 



The imager also supports the use of an automated scoring routine which utilizes 
adjective annotation parameters defined in the database and associates a numerical 
score to the appropriate adjective annotation. The scoring routine first assigns a set of 
numerical values to the fused data component of the image in question. These values 
are then related or mapped back to the adjective scoring system customized by the 
user. . The adjective scoring system consists of a noun description and an adjective 
modifier (e.g., crystal / needles), which are related back to the numerical score by a 
map maintained in the database. Thus the automated scoring system can return 
numerical scores as well as associated adjective scores. 

The imager has the ability automatically find drops on hanging drop-type plates. 
Whsa a hanging drop image is acquired the steps shown in Fig. 7 are performed to 
determine the bounding box of a drop. Tliis allows the imager automation to zoom in 
and maximize the size of the drop in the image. In particular, after an image is 
acquired, an Eigen process is performed on it to generate an eigen image. The eigen 
image is masked and edge-detected. The edge image is filtered through a median 
filter to remove noise and the filtered image is compared with the edge image to 
determine a boundary of a drop in accordance wifli a predetermined threshold. The 
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image in the determined boundary is then despeckled to form a bonndmg box drop 
image. 

The imager database allows the tracking of all interactions of the imager with crystal 
plates. The first time a plate type is loaded into the imager by a user, the user will 
define it for subsequent use by other plates of similar construction. Physical geometry 
information such as nimiber of wells along each axis, well pitch of each axis, nimiber 
of satellite positions, offsets to satelUte positions, plate height, and plate style 
(Linbro/SBS) is stored in the database. In addition, certain camera settings related to 
exposvure, gain, focus, and polarization are associated with each satellite well defined 
on the plate type and stored in the database. 

Once plate types are defined, the system can accept any plates of that plate type for 
imaging. Plates are identified by a unique label or barcode in the database. After 
loading and either scanning or entering a plate label or barcode, the imager will look 
up the plate in the database. If it exists, it will load the plate type information that this 
plate is based on. Entry of new plates will cause the imager to prompt the user to 
select an existing plate type or define a new plate type. After the plate is associated 
with an existing plate type, the user can begin imaging it Each time that a user 
initiates the imaging process, a new image evmt record is created in the system. 
Image events group images of sateUite positions on plates by the date they were 
acquired. The database records the path to the satellite position, image event, image 
file path on disk, milliineters/pixel, polarization angle, image type (Bright Field, Dark 
Field, or Polarized), and image formats (JPG, BMP, PNG, TIF). Multiple images of 
the same or diflfering types may be acquired for a particular satellite during the same 
image event. Once image acquisition is complete, the resulting images may be 
annotated with notes or scored via a set of descriptors and modifiers that each 
customer defines. Annotations occur on a per image event satellite position basis- 
taking all images acquired during that event on that satellite position into accoimt. 
The database allows multiple descriptors and modifiers to be selected for each 
annotation. In addition, multiple scoring methods are supported (manual, automatic, 
hght scatter) by the database. 



The user has the option to store eitiier all images, non-clear images, images with 
crystal detection hits or any combination thereof. The settings for image storage are 
stored in the database and can be filtered by the user to obtain the desired results for 
that particular user. The images themselves are not stored in the database, but rather 
only the image locations. This allows the database size to remain small and 
manageable while still providing a link to the image via a single interface for fixture 
reference and possible scoring. The images for a plate for a particular event are 
located in the same directory that is named for the image event. The filepath is 
recorded and maintained in the database. It has been found that maiataining the 
images external to the protein crystalHzation database allows the database size to be 
managed to a smaller size incireasing efficiency of search and query algorithms. 



The imager also supports an automated plate scheduling algorithm. The plate 
schedules are associated with the plate barcode in the database. The imager scans the 
database to register the next plate scheduled to run in the imager. The plate schedules 
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can be superceded by manually introducing a new plate to image at which time the 
plate scheduled to run will be delayed and run after the manual plate is completed. 
The scheduling algorithm also supports multiple resource management when more 
than one imager resource is available to run a plate and multiple plates are vying for 
imager time. When more tlian one plate is scheduled to run on the imager at a single 
time the scheduling system supports a conflict resolution management system that 
arbitrates conflicts and reorders schedules dependent on user derived rules. 



The imager has the ability to support all common format experiments used in protein 
crystallization today. The results are all stored in a common database regardless of 
the experiment type. The imager requires no definition of experiment type. The 
imager only requires plate type to run all experiment types. Various experiment 
types are handled automatically without requiring user intervention to setup the 
imager to accommodate each type. The following protein crystallization experiment 
types are supported: 

Hanging Drop 

These experiments can be conducted on a variety of plate type (Linbro, VDX, and 
Neuroprobe to a name a few). The imager allows for miiltiple drops per deep well 
and supports automated drop finduig for folly automated image acquisition. 

Sitting Drop 

The imager will support any sitting drop plate on the market today with any well and 
satellite well configuration that can be made on an SBS format plate. It can support 
very dense experiment plates 384 and 1536 as well as large format 24, 48 and 96 well 
plates. 

Microbatch 

The imager will support all common microbatch plate setup formats without requiring 
any special purpose fixturing or user intervention to automatically image. 

Alternative Formats 

The imager also can support sandwich drop experiment types and has provisions built 
into the nest to accommodate adapter plates necessary to support foture experiment 
types such as chip and micro slide based experiments. 

Temperature Range (0° to 40° C) 

The imager can operate across a wide range of enviroranental conditions that are 
common to protein crystalHzation environments in labs today such as fixed humidity, 
low temperature, and overpressure conditions. The temperature range supported by 
the imager is 0° to 40° C. 

The invention having been thus described, it will be apparent to those skilled in the art 
that the same may be varied in many ways without departing finm the spirit and scope 
of the invention. For example, while the preferred embodiment of the invention has 
been described in conjunction with protein crystallization experiments, the imager of 
the invention can be used with any type of crystallization or precipitation experiment 
wherein many different conditions must be configured in order to discover optimal 
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parameter values for crystal or precipitate formation. In the claims which follow, 
"crystaUization" will refer to crystallization or precipitation. Any and all such 
modifications as would be apparent to one skilled in the art are intended to be 
included witliin the scope of the following claims. 
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What Is Claimed : 

1. An apparatus that automatically captures stores and analyzes images of 
crystallization experiments contained in a plurality of crystallization plates, 
comprising: 

a plate nest capable of accommodating protein crystalhzation plates of a plurality 
of different types, said plates each including a plurality of ciystallization weUs each 
including a particular crystallization experiment; 

image acquisition optics, including an objective lens and an image capturing 
device, for focusiag an image of a crystallization well positioned xmder said objective 
lens and electronically capturing said focused image; 

a light source including a bright field illumination device and a dark field 
illumination device, each of said bright field and dark field illumination devices being 
independently energized to iutroduce light into a crystal image well to enable capture 
of separate bright field and dark field images of said crystal image well by said image 
acquisition optics; 

a nest positioning controller for moving the position of said plate nest with respect 
to said image acquisition optics to align various selected wells with said objective lens 
for imaging of the content of said wells; 

a database for storing experiment information associated with each of said 
plurality of crystalhzation plates, said experiment information including 
identification of specific crystal forming parameter values, and wherein each of said 
plurality of crystalhzation plates is identified in said database by a unique 
identification code; and 

a crystallization imaging controller for controlling crystallization imaging by 
retrieving said experiment information for each crystallization plate inserted into said 
apparatus, and controlling said nest positioning controller and said image acquisition 
optics in accordance with said retrieved experiment information. 

2. An apparatus according to claim 1, wherein said crystallization experiments 
comprise protein crystallization experiments, and said crystallization imaging plates 
each contain a protein samples combined with varying amounts of protein 
crystallization inducing catalyst material. 

3. An apparatus according to claim 1, fiirther comprising a transmitter 
polarization filter for polarizing Ught fi:om said hght source to a crystal image well to 
provide a polarized image of said crystal image well for capture by said image 
acquisition optics, 

4. An apparatus according to claim 1, wherein multiple images of individual 
crystal image wells captured by said image acquisition optics under varying lighting 
and/or focusing conditions are processed for forming a single fused image containing 
partial data fiom at least two of said multiple images. 

5. An apparatus according to claim 3, wherein said experiment information 
includes polarization angle information, and said controller controls the polarization 
angle of said polarization filter in accordance with said retrieved polarization angle 
information. 
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6. An apparatus according to claim 5, wherein said experiment information 
includes multiple polarization angles for at least one crystal image well, and said 
controller controls said polarization filter and said image acquisition optics to capture 
multiple images of said crystal image well in accordance with said retrieved multiple 
polarization angle information. 

7 An apparatus according to claim 1, wherein said apparatus analyzes light 
attenuation data from said image acquisition optics for an image well ohtained over a 
predetermined period of tune to calculate a tuibidity value for said image well. 

8 An apparatus according to claim 1, wherein said apparatus analyzes light 
attenuation data from a turbidity detection light detector for an image well obtained 
over a predetemiined period of time to calculate a turbidity value for said image well 

9. An apparatus according to claim 1, wherein crystalUzation plates are loaded and 
unloaded into said plate nest by a robotic plate loader/unloader. 

10. An apparatus according to claim 1, wherein said plate nest further comprises an 
automatic plate centering cam mechanism that ahgns a plate mserted into said plate 

20 nest into an appropriate ahgmnent position subsequent to msertion of said plate into 
said plate nest. 

11. An apparatus according to claim 10, wherein said automatic plate aUgnment 
mechanism automatically aligns an inserted plate. 

12. An apparatus accordmg to claim 1, wherein said image capture device 
comprises a CCD camera, said optics further including a color filter for enabUng true 
color images to be obtained using said CCD camera. 

30 13. An apparatus according to claim 3, wherein said image capture device 

comprises a CCD camera, said optics further including a color filter for enabling true 
color images to be obtained using said CCD camera, and wherein said color filter also 
fimctions as an analyzer polarization filter operatmg in conjunction with said 
transmitter polarization filter to form a polarization system for allowing polarized 

35 light from said crystallization plate to be imaged. 

14 An apparatus according to claim 1, wherem said nest positioning controller 
moves said plate nest in X and Y axis directions in a plane perpendicular to said 
objective lens, and further moves said plate nest m a Z axis direction toward and away 
40 from said objective lens, said apparatus further comprising a Z axis pressure sensor 

for sensmg the presence of a crystallization plate against said image acquisition optics 
and transmitting a signal to said nest positioning controller to cause said nest 
positioning controller to stop movement of said plate nest toward said objective lens 
in said Z axis upon detection of the presence of said crystaUization plate. 

45 

15. An apparatus according to claim 1 , wherein said crystallization plates comprise 
SBS type plates. 

16. An apparatus according to claim 1 , wherein said crystallization plates 
comprise Linbro type plates. 

50 
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17. An apparatus that automatically captures, stores and analyzes images of 
crystallization experiments contained in a plurality of crystallization plates, 
comprising: 

a plate nest capable of accommodating protein crystallization plates of a plurality 
of different types, said plates each including a plurality of crystallization wells each 
including a particular crystallization experiment; 

image acquisition optics, including an objective lens and an image capturing 
device, for focusing an image of a crystallization well positioned under said objective 
lens and electronically capturing said focused image; 

a light source including a bright field illumination device and a dark field 
illmnination device, each of said bright field and dark field illumination devices being 
independently energized to introduce light into a crystal image well to enable capture 
of separate bright field and dark field images of said crystal image well by said image 
acquisition optics; 

a transmitter polarization filter located between said light source and said plate 
nest, for polarizing light fi-om said light source at a desired polarization angle; 

said image acquisition optics finther including an analyzer polarization filter 
operating in conjunction with said transmitter polarization filter as a polarization 
system for allowing polarized light firom said crystallization plate to be imaged; 

wherein multiple images of individual crystal image wells captured by said image 
acquisition optics under varying lighting and/or focusing conditions are processed for 
forming a single fiised image containing partial data firom at least two of said multiple 
images. 

1 8. An apparatus according to claim 1 , wherein said bright field illumination device 
comprises a plurality of light-emitting diodes. 

19. An apparatus according to claim 1, wherein said dark field illumination device 
comprises a plurality of light-emitting diodes. 

20. An apparatus according to claim 17, wherein said bright field illumination device 
comprises a plurality of light-emitting diodes, 

21. An apparatus according to claim 17, wherein said dark field illumination device 
comprises a plurality of light-emitting diodes. 

22. A method of analyzing crystallization experiments contained in a crystallization 
plate, comprising the steps of: 

illuminating a crystallization experiment well of a crystallization plate using a 
brigjit field illummation device to obtain a bright field illumination of said well and 
capturing an image of said well under bright field illumination; 

illuminating said crystallization experiment well using a dark field illumination 
device to obtain a dark field illumiuation of said well and capturing an image of said 
well under dark field illumination; and 

processing said bright field image and said dark field image of said well to 
determine crystallization results of said experiment. 

23. A method as set forth in claim 22, fiirther comprising the step of illuminating 
said crystallization experiment well using polarized light and capturing an image of 
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said weU under polarized Ught iUumination, and wherein said processing step further 
comprises processing said polarized light image. 

24 A method as set forth in claim 22, wherein said processing step comprises the 
st^ of fusing said bright field and dark field unages according to a data fusmg 
algorithm. 

25 A method as set forth in claim 23, wherein said processing step comprises the step 
of fiismg said bright field, dark field, and polarized Ught images accordmg to a data 
fusing algorithm. 
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